In many multi-camera video processing pipelines, the images produced by the imagers (raw images) are warped before being further processed in the pipeline. The warping is done so that the resulting images have useful geometric properties, such as being distortion-free, having square pixels, having aligned epipolar lines etc. These warped images may be referred to as “rectified images”.
The warping parameters that allow these properties to hold depend on the configuration of the cameras including: optical properties of the individual cameras (e.g. focal length, principal point and distortion parameters) and/or on the relative positions of the cameras, and/or the positions of the cameras in the scene. The warping parameters may be typically computed prior to operating the multi-camera system, using a calibration process during which the cameras may be placed in one or more controlled environments, often viewing a known calibration target. The correct operation of a multi-camera system may be dependent on using appropriate warping parameters.
If the optical properties of a camera change or if the camera position relative to the other camera(s) change, then the warping parameters may have to be recomputed. Otherwise, the assumed optical properties of the warped images may not hold any longer and the functionality of the multi-camera system may be compromised. It may then be necessary to find new warping parameters that ensure the images rectified with these new parameters have the desired geometric properties. This may be done by running the calibration process again, which would require interrupting the operation of the multi-camera system and possibly manipulating the cameras physically. Interruptions like this are not desirable because they render the system less practical to use and cause increased operational costs.
Self-calibration of single- and multi-camera systems is sometimes possible. Self-calibration refers to the computation of some or all warping parameters in uncontrolled environments, e.g., without the use of a fixed calibration target. These methods are generally aimed at cameras which do not hold fixed warping parameters because they are designed with intentionally mobile optical elements, such as stereo cameras with variable vergence or cameras with zoom lenses. On these systems, self-calibration is generally performed constantly, at the cost of computational resources, frame rate and/or latency.
For a multi-camera system which is designed with fixed optical elements (e.g., lens and imagers that are held rigidly with respect to each other, lens focal lengths that are fixed, and rigid relative positioning between imaging units), the warping parameters are not expected to change dramatically during use. As a consequence, constantly performing self-calibration is not useful and would incur added cost in computational resource, frame rate, and/or latency. However, physical shock to the multi-camera system, or other environmental effects such as a large temperature change, could introduce changes to the system which would require some warping parameters to be recomputed.
In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears.